Data processing apparatus, operation accepting method, and non-transitory computer-readable recording medium encoded with browsing program

ABSTRACT

A data processing apparatus includes a browsing portion to execute a process of the type corresponding to a gesture event, a mouse event generating portion to generate a mouse event including positional information indicating the position detected by a position detecting portion, and a gesture event determining portion to determine one of gesture events on the basis of the mouse event(s). The browsing portion includes a display image generating portion operable, when content includes a description for displaying an embedded image generated as an application program is executed, to generate a display image with the embedded image arranged in an embedded area, and a selecting portion operable, when the position indicated by one of the mouse event(s) from which the gesture event has been determined falls within the embedded area, to select either the gesture event or the mouse event(s) on the basis of the type of the gesture event.

This application is based on Japanese Patent Application No. 2012-260148filed with Japan Patent Office on Nov. 28, 2012, the entire content ofwhich is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a data processing apparatus, anoperation accepting method, and a non-transitory computer-readablerecording medium encoded with a browsing program. More particularly, thepresent invention relates to a data processing apparatus provided with amulti-touch-capable touch panel, an operation accepting method performedby the data processing apparatus, and a non-transitory computer-readablerecording medium encoded with a browsing program for causing a computerto perform the operation accepting method.

2. Description of the Related Art

Some recent data processing apparatuses, typified by multi-functionperipherals (MFPs), are provided with a multi-touch-capable touch panelas a user interface. The multi-touch-capable touch panel can detect aplurality of positions simultaneously designated by a user. On the otherhand, some MFPs allow application programs to be installed therein. Auser can customize an MFP by installing an application program accordingto the type of usage. Of these application programs, while some programscan accept a multi-touch operation, others cannot accept the multi-touchoperation and can accept a single-touch operation. Therefore, in thecase where a plurality of application programs are executed at the sametime, it is necessary to switch between the mode of accepting useroperations as multi-touch operations and the mode of accepting useroperations as single-touch operations, for each of the applicationprograms.

For example, Japanese Patent Laid-Open No. 2009-211704 describes amethod for handling mouse events in a multi-touch device. The methodincludes the steps of: displaying one or more views; executing one ormore software elements, each associated with a particular view;associating a multi-touch flag or an exclusive touch flag with eachview; receiving one or more touches on the one or more views; and, onthe basis of the value(s) of the multi-touch or exclusive touch flag(s),selectively sending one or more mouse events, each describing thecorresponding received touch, to the one or more software elementsassociated with the one or more views on which the one or more toucheswere received.

Some application programs, however, output an image (view) whichincludes an area accepting multi-touch operations and an area acceptingsingle-touch operations. In the conventional method, the mode ofaccepting user operations as multi-touch operations and the mode ofaccepting them as single-touch operations are switched depending upon aview. Therefore, the conventional method cannot address the case where asingle view (image) includes the area accepting multi-touch operationsand the area accepting single-touch operations.

SUMMARY OF THE INVENTION

An aspect of the present invention provides a data processing apparatusincluding a display portion capable of displaying an image, a positiondetecting portion capable of detecting one or more positions designatedby a user on a display surface of the display portion, and a controlportion, wherein the control portion includes: a browsing portion toexecute a browsing program, externally acquire content described in amarkup language, and display on the display portion at least a part of adisplay image corresponding to the acquired content, and furtheroperable, in response to input of one of a plurality of gesture eventscorresponding respectively to a plurality of types of processesdetermined by the browsing program, to perform the one of the pluralityof types of processes that corresponds to the input gesture event; amouse event generating portion operable, in response to detection of oneor more positions by the position detecting potion, to generate one ormore mouse events each including a piece of positional informationindicating the corresponding one of the one or more positions; and agesture event determining portion to determine one of the plurality ofgesture events on the basis of a set of one or more mouse eventsgenerated by the mouse event generating portion; wherein the browsingportion includes a display image generating portion operable, in thecase where the content includes a description for displaying an embeddedimage to be generated as an application program different from thebrowsing program is executed, to generate a display image in which theembedded image generated as the application program has been executed isarranged in an embedded area determined by the content, and a selectingportion operable, in the case where the position indicated by one of theone or more mouse events on the basis of which the gesture event hasbeen determined by the gesture event determining portion falls withinthe embedded area, to select either the gesture event determined by thegesture event determining portion or the set of the one or more mouseevents on the basis of which the gesture event has been determined, onthe basis of the type of the determined gesture event.

Another aspect of the present invention provides an operation acceptingmethod performed by a data processing apparatus, the data processingapparatus including a display portion capable of displaying an image anda position detecting portion capable of detecting one or more positionsdesignated by a user on a display surface of the display portion,wherein the method include: a browsing step of executing a browsingprogram, externally acquiring content described in a markup language,and displaying on the display portion at least a part of a display imagecorresponding to the acquired content, and further, in response to inputof one of a plurality of gesture events corresponding respectively to aplurality of types of processes determined by the browsing program,performing the one of the plurality of types of processes thatcorresponds to the input gesture event; a mouse event generating stepof, in response to detection of one or more positions by the positiondetecting potion, generating one or more mouse events each including apiece of positional information indicating the corresponding one of theone or more positions; and a gesture event determining step ofdetermining one of the plurality of gesture events on the basis of a setof one or more mouse events generated in the mouse event generatingstep; wherein the browsing step includes a display image generating stepof, in the case where the content includes a description for displayingan embedded image which is generated as an application program differentfrom the browsing program is executed, generating a display image inwhich the embedded image generated as the application program has beenexecuted is arranged in an embedded area determined by the content, anda selecting step of, in the case where the position indicated by one ofthe one or more mouse events on the basis of which the gesture event hasbeen determined in the gesture event determining step falls within theembedded area, selecting either the gesture event determined in thegesture event determining step or the set of the one or more mouseevents on the basis of which the gesture event has been determined, onthe basis of the type of the determined gesture event.

Yet another aspect of the present invention provides a non-transitorycomputer-readable recording medium encoded with a browsing programperformed by a computer which controls a data processing apparatus, thedata processing apparatus including a display portion capable ofdisplaying an image, a position detecting portion capable of detectingone or more positions designated by a user on a display surface of thedisplay portion, a mouse event generating portion operable, in responseto detection of one or more positions by the position detecting potion,to generate one or more mouse events each including a piece ofpositional information indicating the corresponding one of the one ormore positions, and a gesture event determining portion to determine oneof the plurality of gesture events on the basis of a set of one or moremouse events generated by the mouse event generating portion, whereinthe program causes the computer to perform a browsing step of executinga browsing program, externally acquiring content described in a markuplanguage, and displaying on the display portion at least a part of adisplay image corresponding to the acquired content, and further, inresponse to input of one of a plurality of gesture events correspondingrespectively to a plurality of types of processes determined by thebrowsing program, performing the one of the plurality of types ofprocesses that corresponds to the input gesture event, wherein thebrowsing step includes a display image generating step of, in the casewhere the content includes a description for displaying an embeddedimage to be generated as an application program different from thebrowsing program is executed, generating a display image in which theembedded image generated as the application program has been executed isarranged in an embedded area determined by the content, and a selectingstep of, in the case where the position indicated by one of the one ormore mouse events on the basis of which the gesture event has beendetermined by the gesture event determining portion falls within theembedded area, selecting either the gesture event determined by thegesture event determining portion or the set of the one or more mouseevents on the basis of which the gesture event has been determined, onthe basis of the type of the determined gesture event.

The foregoing and other features, aspects, and advantages of the presentinvention will become more apparent from the following detaileddescription of the present invention when taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of an MFP according to an embodiment of thepresent invention;

FIG. 2 is a plan view showing an example of an operation panel;

FIG. 3 is a block diagram schematically showing an example of thehardware configuration of the MFP;

FIG. 4 shows an example of the software architecture of a CPU includedin the MFP;

FIG. 5 is a block diagram showing, by way of example, the functions ofthe CPU included in the MFP, together with information stored in a HDD;

FIG. 6 shows an example of a master table;

FIG. 7 is a flowchart illustrating an example of the flow of operationsperformed by a user and processing performed by the MFP;

FIG. 8 is a flowchart illustrating an example of the flow of a gestureevent determining process;

FIG. 9 is a flowchart illustrating an example of the flow of a browsingprocess;

FIG. 10 shows an example of a display image; and

FIGS. 11 to 13 show examples of a display area in the display image.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The preferred embodiments of the present invention will be describedbelow in conjunction with the drawings. In the following description,the same or corresponding parts are denoted by the same referencecharacters. Their names and functions are also the same. Thus, adetailed description thereof will not be repeated.

FIG. 1 is a perspective view of an MFP according to an embodiment of thepresent invention. FIG. 2 is a plan view showing an example of anoperation panel. Referring to FIGS. 1 and 2, an MFP 100 includes: adocument reading portion 130 for reading a document; an automaticdocument feeder 120 for feeding a document to document reading portion130; an image forming portion 140 for forming an image on a sheet ofpaper or the like on the basis of image data which is output fromdocument reading portion 130 that has read a document; a paper feedingportion 150 for feeding a sheet of paper to image forming portion 140;and an operation panel 160 serving as a user interface.

Operation panel 160 is arranged on an upper surface of MFP 100.Operation panel 160 includes: a display portion 161, a touch panel 165,and a hard key portion 167. Display portion 161 is, for example, aliquid crystal display (LCD), and displays an instruction menu for auser, information about acquired image data, and so on. Hard key portion167 includes four hard keys 167A to 167D having the characters “BOX”,“FAX”, “COPY”, and “ECO” respectively displayed thereon. Touch panel 165is a multi-touch-capable touch panel which is disposed on an upper orlower surface of display portion 161, and detects a position designatedby a user on the display surface of display portion 161. The useroperations of designating touch panel 165 include a multi-touchoperation and a single-touch operation. In the multi-touch operation, auser uses a plurality of fingers to designate a plurality of locationson touch panel 165 simultaneously. In the single-touch operation, a useruses one finger to designate one location at a time.

Automatic document feeder 120 automatically feeds a plurality ofdocuments set on a document feed tray, one by one, to a predetermineddocument reading position set on a platen glass of document readingportion 130, and outputs the document, the image of which has been readby document reading portion 130, onto a document output tray. Documentreading portion 130 includes an optical source which irradiates adocument placed on the document reading position with light and aphotoelectric conversion element which receives light reflected from thedocument, and scans a document image having a size corresponding to thatof the document. The photoelectric conversion element converts thereceived light into image data made up of electric signals, and outputsthe image data to image forming portion 140. Paper feeding portion 150feeds a sheet of paper, stored in a paper feed tray, to image formingportion 140.

Image forming portion 140 forms an image using well-knownelectrophotography. Image forming portion 140 performs various kinds ofdata processing such as shading compensation on image data received fromdocument reading portion 130 and, on the basis of the processed imagedata, or on the basis of externally received image data, forms an imageon a sheet of paper fed by paper feeding portion 150.

FIG. 3 is a block diagram schematically showing an example of thehardware configuration of the MFP. Referring to FIG. 3, MFP 100 includesa main circuit 110. Main circuit 110 includes: a central processing unit(CPU) 111; a communication interface (I/F) portion 112; a read onlymemory (ROM) 113; a random access memory (RAM) 114; a hard disk drive(HDD) 115 as a mass storage; a facsimile portion 116; and an externalstorage device 117. CPU 111 is connected with automatic document feeder120, document reading portion 130, image forming portion 140, paperfeeding portion 150, and operation panel 160, and is responsible foroverall control of MFP 100.

ROM 113 stores a program to be executed by CPU 111 or data necessary forexecution of the program. RAM 114 is used as a work area when CPU 111executes a program. Further, RAM 114 temporarily stores read imagescontinuously transmitted from document reading portion 130.

Facsimile portion 116 is connected to public switched telephone networks(PSTN), and transmits facsimile data to or receives facsimile data fromthe PSTN. Facsimile portion 116 stores the received facsimile data inHDD 115. Alternatively, facsimile portion 116 converts the data intoprint data which can be printed in image forming portion 140, andoutputs the print data to image forming portion 140, thereby causingimage forming portion 140 to form an image on a sheet of paper on thebasis of the facsimile data received by facsimile portion 116. Further,facsimile portion 116 converts the data read by document reading portion130 or the data stored in HDD 115 into facsimile data, and transmits thefacsimile data to a facsimile machine connected to the PSTN.

Communication I/F portion 112 is an interface for connecting MFP 100 toa network. Communication I/F portion 112 communicates with anothercomputer connected to the network, using a communication protocol suchas the transmission control protocol (TCP) or the file transfer protocol(FTP). It is noted that the protocol for communication is notparticularly limited; any protocol can be used. The network to whichcommunication I/F portion 112 is connected is, for example, a local areanetwork (LAN). It may be connected in a wired or wireless manner. Thenetwork is not necessarily the LAN; it may be a wide area network (WAN),a network using the public switched telephone networks (PSTN), or thelike. Furthermore, the network is connected to the Internet, so that itis possible for MFP 100 to communicate with a computer, such as aserver, connected to the Internet.

External storage device 117, which is controlled by CPU 111, is mountedwith a compact disc read-only memory (CD-ROM) 118 or a semiconductormemory. CPU 111 is capable of accessing CD-ROM 118 or the semiconductormemory via external storage device 117. CPU 111 loads the programrecorded on CD-ROM 118 or the semiconductor memory mounted on externalstorage device 117, into RAM 114 for execution. It is noted that theprogram executed by CPU 111 is not restricted to the program recorded onCD-ROM 118 or the semiconductor memory. CPU 111 may load a programstored in HDD 115, into RAM 114 for execution. In this case, via thenetwork to which communication I/F portion 112 is connected, anothercomputer connected to the network may rewrite the program stored in HDD115 of MFP 100, or additionally write a new program therein. Further,MFP 100 may download a program from another computer connected to thenetwork, and store the program in HDD 115. As used herein, the “program”includes, not only the program which CPU 111 can execute directly, butalso a source program, a compressed program, an encrypted program, andso on.

It is noted that the medium for storing a program executed by CPU 111 isnot restricted to CD-ROM 118. It may be an optical disc (magneto-optical(MO) disc, mini disc (MD), digital versatile disc (DVD)), an IC card, anoptical card, or a semiconductor memory such as a mask ROM, an erasableprogrammable ROM (EPROM), an electrically erasable and programmable ROM(EEPROM), or the like.

Operation panel 160 includes display portion 161 and an operationportion 163. Display portion 161 is a display such as a liquid crystaldisplay (LCD) or an organic electro-luminescence display (ELD), anddisplays an instruction menu for a user, information about acquiredimage data, and so on. Operation portion 163 includes touch panel 165and hard key portion 167 made up of a plurality of hard keys. The hardkeys included in hard key portion 167 each include a contact switch,which is connected to CPU 111. Each hard key, when depressed by anoperation user, closes its contact to close a circuit connected to CPU111. Each hard key closes the circuit while being depressed by anoperation user who operates MFP 100, whereas it opens the circuit whilenot being depressed by the operation user.

When a plurality of keys included in hard key portion 167 are depressed,operation portion 163 accepts input data, such as instructions,characters, and numerical characters, corresponding to the depressedkeys. Touch panel 165 is disposed on an upper or lower surface ofdisplay portion 161, and outputs the coordinates of a positiondesignated by the operation user to CPU 111. Touch panel 165 detects theposition designated by the operation user with his/her finger or astylus pen, and outputs the coordinates of the detected position to CPU111. Touch panel 165 is a multi-touch-capable touch panel. When a userinputs a multi-touch operation, touch panel 165 outputs to CPU 111 aplurality of sets of coordinates corresponding respectively to theplurality of positions designated simultaneously by the user. When auser inputs a single-touch operation, touch panel 165 outputs to CPU 111a set of coordinates corresponding to the single position designated bythe user.

Touch panel 165 preferably has a size equal to or greater than that ofthe display surface of display portion 161. As touch panel 165 isdisposed on the surface of display portion 161, when the operation userdesignates a position on the display surface of display portion 161,touch panel 165 outputs the coordinates of the position that theoperation user has designated on the display surface of display portion161, to CPU 111. Touch panel 165 may be, for example, of a resistivetype, a surface acoustic wave type, an infrared ray type, anelectromagnetic induction type, or a capacitance type, although it isnot limited to these types.

Operations that can be accepted by operation portion 163 include anoperation of depressing any of the keys included in hard key portion167, and a single-touch operation and a multi-touch operation ofdesignating position(s) on touch panel 165. Further, hard key portion167 includes four hard keys 167A to 167D having the characters “BOX”,“FAX”, “COPY”, and “ECO” respectively displayed thereon. Three hard keys167A, 167B, and 167C with the characters “BOX”, “FAX”, and “COPY”,respectively, are process switching keys assigned with process switchingoperations for designating switching of the process to be executed byMFP 100. Hard key 167D with the characters “ECO” is a mode switching keyassigned with a mode switching operation for designating switching ofthe mode of MFP 100 from a normal mode to a power saving mode in whichless electricity is consumed. When hard key 167A, 167B, or 167C isdepressed, operation portion 163 accepts the corresponding processswitching operation. When hard key 167D is depressed, operation portion163 accepts the mode switching operation.

FIG. 4 shows an example of the software architecture of the CPU includedin the MFP. Referring to FIG. 4, in CPU 111, an operating system (OS)layer is formed as the bottom layer, and an MFP process layer is formedabove the OS layer. Above the MFP process layer, an application platform(PF) layer is formed, and an application layer is formed above theapplication PF layer.

A task for CPU 111 to execute an OS program belongs to the OS layer. Thetask belonging to the OS layer performs the process of controllinghardware resources of MFP 100. Here, the hardware resources includecommunication I/F portion 112, ROM 113, RAM 114, HDD 115, facsimileportion 116, external storage device 117, automatic document feeder 120,document reading portion 130, image forming portion 140, paper feedingportion 150, and operation panel 160. The task belonging to the OS layercontrols the hardware resources in accordance with an operating commandinput from the MFP process layer. Further, the OS layer shares, with theMFP process layer, a plurality of pieces of operation identificationinformation for respectively identifying a plurality of types ofoperations that can be accepted by operation portion 163 which is one ofthe hardware resources. When operation portion 163 detects an operationperformed by an operation user, the task belonging to the OS layeroutputs a piece of operation identification information for identifyingthe detected operation, to the MFP process layer.

A task for CPU 111 to execute an application program belongs to theapplication layer. In the case where two or more application programsare installed into MFP 100, two or more tasks of executing therespective application programs may belong to the application layer.

The task of executing an application program performs a plurality oftypes of processes determined by the application program. The pluralityof types of processes include a process of causing a task belonging tothe MFP process layer to perform a process to be performed in the MFPprocess layer. In the case of causing the task belonging to the MFPprocess layer to perform a process, the task of executing theapplication program outputs an application command. The applicationcommand is a command which has been released as an application programinterface (API). This facilitates creation of an application program forcausing MFP 100 to perform a process. Releasing something means that athird party other than the manufacturer of MFP 100 can use it. The thirdparty is able to use the application commands to develop an applicationprogram that can be installed into MFP 100.

Further, the task of executing the application program specifies aprocess to be performed, from among the plurality of types of processesdetermined by the application program, on the basis of an operationinput by the operation user who operates MFP 100. The task of executingthe application program then performs the specified process. Theoperation input by the operation user operating MFP 100 is accepted inthe OS layer when the operation user operates operation portion 163 toinput the operation.

The application PF layer, located between the application layer and theMFP process layer, has belonging thereto a task of arbitrating aplurality of tasks belonging to the application layer and alsocontrolling the application commands output from the plurality of tasksbelonging to the application layer. More specifically, the applicationPF layer accepts application commands output respectively from the tasksbelonging to the application layer, converts the accepted applicationcommands into internal commands in accordance with a predetermined rule,for standardization, and outputs the internal commands to the MFPprocess layer. This can address the case where a plurality ofapplication programs are different in version, or the like. Further, theapplication PF layer brings one of the tasks belonging to theapplication layer into a currently selected state. The application PFlayer outputs the operation which has been accepted by the OS layer andis input from the task belonging to the MFP process layer, to the taskthat has been set to the currently selected state from among the tasksbelonging to the application layer.

The operations input by a user and accepted in the OS layer include asingle-touch operation in which the user uses one finger to designate aposition on touch panel 165 and a multi-touch operation in which theuser uses two or more fingers to designate positions on touch panel 165simultaneously. Each of the single-touch operation and the multi-touchoperation is detected as a mouse event by the operating system, and isinput from the application PF layer into the currently selected taskamong the tasks belonging to the application layer. Further, one or moremouse events are converted into a gesture event in the MFP processlayer, and the gesture event is input from the application PF layer intothe currently selected task among the tasks belonging to the applicationlayer. To this end, the application layer shares, with the MFP processlayer, one or more gesture events which can be accepted by anapplication program, and the task of executing the application programreceives, from the task belonging to the application PF layer, one ormore gesture events acceptable by the application program. The task ofexecuting the application program performs one or more of the pluralityof types of processes that correspond(s) to the gesture event(s) or themouse event(s).

Application commands are associated with internal commands in advance.For example, a commands correspondence table which associatesapplication commands with internal commands may be stored. Oneapplication command may correspond to one internal command, or oneapplication command may correspond to a set of two or more internalcommands. Further, two or more application commands of differentversions may correspond to one internal command or to a set of two ormore internal commands. This can address the case where applicationprograms are of different versions. The internal command is a commandwhich depends upon the hardware resource(s) in MFP 100. Internalcommands are not released usually, although they may be released.

The MFP process layer, located between the application PF layer and theOS layer, has belonging thereto a task for CPU 111 to execute an MFPprocess program. The MFP process layer converts an internal commandoutput from a task belonging to the application PF layer, into anoperating command which can be interpreted by a task belonging to the OSlayer, and outputs the operating command to the task belonging to the OSlayer for controlling the hardware resource(s). Although an internalcommand may actually be converted into one or more operating commandswhich can be executed by a task belonging to the OS layer, for theconvenience sake, it is here assumed that one internal command isconverted into one operating command executable by the task belonging tothe OS layer.

While the application programs cause MFP 100 to perform the browsingprocess, copying process, scanning process, printing process, facsimiletransmitting/receiving process, data transmitting/receiving process, andso on, the following description will focus on the part related to theprocess of controlling operation panel 160, among the processes theapplication programs cause CPU 111 to perform.

FIG. 5 is a block diagram showing, by way of example, the functions ofthe CPU included in the MFP, together with information stored in theHDD. The functions shown in FIG. 5 are implemented by CPU 111 includedin MFP 100 as CPU 111 executes programs stored in ROM 113, HDD 115, orCD-ROM 118. Specifically, the functions are implemented by CPU 111 asCPU 111 executes an OS program, an MFP process program, an operationaccepting program, and an application program. Shown here are thefunctions of CPU 111 implemented when a first application program and asecond application program are installed in MFP 100 as the applicationprograms.

Referring to FIG. 5, CPU 111 includes an application portion 51, anarbitration portion 53, an operation converting portion 55, and anoperating portion 57. Application portion 51 belongs to the applicationlayer in the software architecture shown in FIG. 4. Application portion51 includes a browsing portion 61 and a secondary executing portion 63.Browsing portion 61 executes a browsing program which corresponds to thefirst application program. Secondary executing portion 63 executes thesecond application program different from the browsing program.

Browsing portion 61 includes a display image generating portion 71, adetermining portion 73, a selecting portion 75, and an instructingportion 77. Display image generating portion 71 downloads from a serverconnected to the Internet a Web page which is content described in amarkup language such as hypertext markup language (HTML). Display imagegenerating portion 71 interprets the Web page, generates a display imagecorresponding to the Web page, and displays at least a part of thedisplay image on display portion 161. The Web page may be a Web pageidentified by the uniform resource locator (URL) which is designated byan operation user who operates MFP 100, or a Web page identified by theURL predetermined by browsing portion 61.

Specifically, display image generating portion 71 outputs an applicationcommand instructing to display on display portion 161 at least a part ofa display image corresponding to the Web page, to arbitration portion53. This causes operating portion 57, which will be described later, tostore the display image into a video RAM (\TRAM) included in displayportion 161, and display at least a part of the display image on displayportion 161. The area, within the display image, to be displayed ondisplay portion 161 can be determined arbitrarily. For example, the areato be displayed on display portion 161 may be determined by default,determined by the Web page, or determined on the basis of the sizes ofthe display image and display portion 161.

Further, when the Web page acquired from the server includes adescription for displaying an embedded image which is generated when thesecond application program different from the browsing program isexecuted, display image generating portion 71 outputs a start-upinstruction to secondary executing portion 63. This causes secondaryexecuting portion 63 to be formed in CPU 111. When the Web page includesa description for displaying an embedded image, display image generatingportion 71 refrains from arranging any image in the embedded area,determined by the Web page, in the display image. The embedded area isan area within a display image in which an embedded image is to bearranged.

The start-up instruction includes positional information indicating theposition of an embedded area in a display image. The start-upinstruction may also include content identification information forspecifying embedded content which is different from the Web page. Thecontent identification information is, for example, a network address,preferably a URL.

Secondary executing portion 63 is started in response to input of astart-up instruction from display image generating portion 71. Secondaryexecuting portion 63 executes a program which is described in anobject-oriented programming language such as Java (registeredtrademark), JavaScript (registered trademark), or the like. Secondaryexecuting portion 63 may execute a second application program differentfrom the browsing program, or may cause a server, which is an externalcomputer connected via a network, to execute the second applicationprogram different from the browsing program.

In the case where secondary executing portion 63 causes a server,connected to the network, to execute the second application program,secondary executing portion 63 may or may not be included in browsingportion 61. The case where secondary executing portion 63 is included inbrowsing portion 61 corresponds, for example, to the case wheresecondary executing portion 63 executes an application program describedin JavaScript (registered trademark). In this case, the applicationprogram described in JavaScript (registered trademark) is describedwithin the browsing program which is the first application program. Thecase where secondary executing portion 63 is not included in browsingportion 61 corresponds, for example, to the case where secondaryexecuting portion 63 executes an application program described in Java(registered trademark). In this case, the application program describedin Java (registered trademark) is a program different from the browsingprogram which is the first application program.

Secondary executing portion 63 acquires an image of content which isdetermined by the second application program executed by the server, asan embedded image. Specifically, secondary executing portion 63transmits to the server a processing request for causing the server toperform a process predetermined by the second application program. Whensecondary executing portion 63 receives from the server an image thatthe server has generated by performing the process in accordance withthe processing request, secondary executing portion 63 acquires thereceived image as the embedded image. In the case where the content tobe displayed is determined by the server, there is no need for theprocessing request to include content identification information.Alternatively, the content may be specified in the Web page, in whichcase secondary executing portion 63 may transmit to the server aprocessing request including the content identification informationincluded in the start-up instruction, to cause the server to process thecontent specified by the content identification information and returnan image of the content to secondary executing portion 63. It is notedthat the server which secondary executing portion 63 causes to performthe processing may or may not be the same as the server from whichbrowsing portion 61 has downloaded the Web page.

In the case where secondary executing portion 63 executes a secondapplication program different from the browsing program, the secondapplication program is an application program that is different from thebrowsing program, which is, for example, a reproducing program forgenerating a still image or a moving image which is the content. In thiscase, secondary executing portion 63 executes the reproducing programwhich is the second application program, and causes CPU 111 to performthe process of reproducing an image of the content specified by thecontent identification information included in the start-up instruction.

Secondary executing portion 63 displays on display portion 161, as anembedded image, an image of the content that has been received from theserver, or an image of the content that has been generated as the secondapplication program has been executed. Specifically, secondary executingportion 63 outputs an application command to arbitration portion 53 soas to cause the embedded image to be arranged in an embedded area in adisplay image. Further, secondary executing portion 63 outputs a displayrequest to browsing portion 61. As a result, the embedded image isstored, by operating portion 57 which will be described later, into theembedded area within the display image which is stored in the VRAMincluded in display portion 161. The application command includes thepositional information included in the start-up instruction.

It is noted that the application programs executed by browsing portion61 and secondary executing portion 63 are not limited to those describedabove. The application program may be a program for customizing a userinterface and the like for causing MFP 100 to perform the copyingprocess, scanning process, printing process, facsimiletransmitting/receiving process, data transmitting/receiving process, andso on.

Arbitration portion 53 and operation converting portion 55 are functionsimplemented by tasks for CPU 111 to execute an operation acceptingprogram. Arbitration portion 53 belongs to the application PF layer inthe software architecture shown in FIG. 4. Operation converting portion55 also belongs to the application PF layer in the software architectureshown in FIG. 4. While it is here assumed that all the functions ofoperation converting portion 55 belong to the application PF layer, itmay be configured such that at least some of the functions of operationconverting portion 55 belong to the application PF layer and the restbelong to the MFP process layer.

In the case where there are a plurality of tasks of executingapplication programs in application portion 51, arbitration portion 53brings one of the plurality of tasks into a currently selected state.Hereinafter, the task that has been set in the currently selected stateby arbitration portion 53 will be referred to as “current task”. In thepresent embodiment, application portion 51 includes browsing portion 61and secondary executing portion 63, and secondary executing portion 63is started by browsing portion 61. Therefore, arbitration portion 53determines browsing portion 61 to be the current task.

Arbitration portion 53 receives an application command output from eachof browsing portion 61 and secondary executing portion 63, and outputsthe application command and application identification information foridentifying the task that has output the application command, tooperation converting portion 55.

In the case where a plurality of application commands are output frombrowsing portion 61 and secondary executing portion 63, arbitrationportion 53 determines the order of the application commands, andsequentially outputs them in the determined order to operationconverting portion 55. For example, in the case where a plurality ofapplication commands cannot be executed at the same time, arbitrationportion 53 waits until operation converting portion 55 finishes theexecution of one application command before arbitration portion 53outputs another application command to operation converting portion 55.In the case where one application command can be executed only after theexecution of another application command is finished, even if the oneapplication command is input earlier than the other application command,arbitration portion 53 outputs the other application command first.

Operation converting portion 55 receives an application command fromarbitration portion 53, and converts the application command, inaccordance with a commands correspondence table, into an internalcommand for standardization. The commands correspondence tableassociates one application command with one or more internal commands.The application commands included in the commands correspondence tablemay include the application commands of the same type but of differentversions. In such a case, each of the application commands of differentversions is associated with one or more internal commands. This enablesinstallation of application programs having application commands ofdifferent versions described therein. When a new application commandappears, the commands correspondence table is updated with a commandscorrespondence table in which the new application command is associatedwith one or more internal commands. This enables installation of anapplication program having the new application command describedtherein.

Further, operation converting portion 55 converts the standardizedinternal command into an operating command, and outputs the operatingcommand to operating portion 57. The operating command is a command thathas been predetermined between operation converting portion 55 andoperating portion 57 and that can be interpreted by operating portion57.

Operating portion 57 is a function implemented by a task for CPU 111 toexecute an OS program. Operating portion 57 belongs to the OS layer inthe software architecture shown in FIG. 4.

Operating portion 57 receives an operating command output from operationconverting portion 55, and controls hardware resources in accordancewith the operating command. Here, display portion 161, touch panel 165in operation portion 163, and HDD 115 will be described as examples ofthe hardware resources. Operating portion 57 controls display portion161 to cause it to display an image. Further, operating portion 57controls HDD 115 to cause it to store data, and also reads data storedin HDD 115. Furthermore, operating portion 57 causes touch panel 165 todetect a position designated by a user, at a prescribed time interval,and acquires positional information from touch panel 165 at a prescribedtime interval.

When an operation user designates a position on the display surface ofdisplay portion 161, touch panel 165 detects the position designated bythe operation user, and outputs the positional information indicatingthe position on the display surface of display portion 161, to CPU 111.When operating portion 57 receives positional information from touchpanel 165, operating portion 57 outputs the positional information tooperation converting portion 55. Operating portion 57 may receive two ormore pieces of positional information simultaneously from touch panel165. For example, when an operation user uses two fingers to designatetwo different positions on the display surface of display portion 161simultaneously, operating portion 57 accepts two pieces of positionalinformation which respectively indicate the two positions designated onthe display surface. When operating portion 57 accepts two or morepieces of positional information simultaneously from touch panel 165,operating portion 57 outputs the two or more pieces of positionalinformation to operation converting portion 55.

Operation converting portion 55 includes an association portion 81, agesture event determining portion 83, a mouse event generating portion85, and a display control portion 87. At the stage when a browsingprogram is installed, association portion 81 associates each of aplurality of types of processes determined by the browsing program withone of a plurality of gesture events. Specifically, association portion81 generates a master table in which a plurality of pieces of processidentification information respectively identifying the plurality oftypes of processes determined by the browsing program are associatedwith corresponding ones of gesture events acceptable by the browsingprogram, and stores the generated master table into HDD 115.

FIG. 6 shows an example of a master table. Referring to FIG. 6, mastertable 91 includes a master record for each of a plurality of pieces ofprocess identification information. Each master record includes a“process” field and a “gesture” field. In the “process” field, processidentification information is set. In the “gesture” field, gestureidentification information for identifying a gesture event is set.

For example, the master record with the process identificationinformation “Page Turning” set in the “process” field has the gestureidentification information “Swipe” set in the “gesture” field. Theprocess specified by the process identification information “PageTurning” is the process of changing the screen to a next page. Thegesture event specified by the gesture identification information“Swipe” corresponds to the user operation of moving a finger, whiledesignating a position on touch panel 165, to the left, right, up, ordown at a speed equal to or higher than a prescribed speed. As thedirection of moving a finger is determined during the operation ofmoving the finger at a speed equal to or higher than a prescribed speed,the gesture event specified by the gesture identification information“Swipe” includes the direction of moving the finger as a parameter.

The master record with the process identification information“Enlargement” set in the “process” field has the gesture identificationinformation “Pinch Out” set in the “gesture” field. The processspecified by the process identification information “Enlargement” is theprocess of zooming in on the screen. The gesture event specified by thegesture identification information “Pinch Out” corresponds to the useroperation of moving at least one of the two locations designated ontouch panel 165 such that the spacing between the two locationsincreases. As the spacing between the two locations is changed duringthe operation of moving the finger(s) to increase the spacing betweenthe two locations, the gesture event specified by the gestureidentification information “Pinch Out” includes, as a parameter, ascaling factor indicating the rate of change of the spacing between thetwo locations.

The master record with the process identification information“Reduction” set in the “process” field has the gesture identificationinformation “Pinch In” set in the “gesture” field. The process specifiedby the process identification information “Reduction” is the process ofzooming out on the screen. The gesture event specified by the gestureidentification information “Pinch In” corresponds to the user operationof moving at least one of the two locations designated on touch panel165 such that the spacing between the two locations decreases. As thespacing between the two locations is changed during the operation ofmoving the finger(s) to decrease the spacing between the two locations,the gesture event specified by the gesture identification information“Pinch In” includes, as a parameter, a scaling factor indicating therate of change of the spacing between the two locations.

The master record with the process identification information “Rotation”set in the “process” field has the gesture identification information“Rotation” set in the “gesture” field. The process specified by theprocess identification information “Rotation” is the process of rotatinga screen to change the direction of the screen. The gesture eventspecified by the gesture identification information “Rotation” isdetected when the user operation of moving at least one of the twolocations designated on touch panel 165 such that the direction of thestraight line connecting the two locations is changed is input. Forexample, a user may input an operation of drawing an arc with one fingerabout the location designated by the other finger. As the direction inwhich the arc is drawn is clockwise or counterclockwise, the gestureevent specified by the gesture identification information “Rotation”includes either the clockwise direction or the counterclockwisedirection as a parameter.

The master record with the process identification information “Scroll”set in the “process” field has the gesture identification information“Flick” set in the “gesture” field. The process specified by the processidentification information “Scroll” is the process of scrolling a screento the left, right, up, or down. The gesture event specified by thegesture identification information “Flick” corresponds to the useroperation of moving a finger, while designating a position on touchpanel 165, to the left, right, up, or down at a speed lower than thethreshold value that is also used for determination of the useroperation corresponding to the gesture event identified by the gestureidentification information “Swipe”. As the direction of moving a fingeris determined during the operation of moving the finger at a speed lowerthan the threshold value, the gesture event specified by the gestureidentification information “Flick” includes the direction of moving thefinger as a parameter.

The master record with the process identification information “SelectContent” set in the “process” field has the gesture identificationinformation “Double Tap” set in the “gesture” field. The processspecified by the process identification information “Select Content” isthe process of selecting content.

Returning to FIG. 5, display control portion 87 converts applicationcommands input from browsing portion 61 or secondary executing portion63, into operating commands for causing display portion 161 to displayan image, and outputs the operating commands to operating portion 57.This causes a display image generated by browsing portion 61 and anembedded image generated by secondary executing portion 63 to be storedinto the VRAM in display portion 161, and at least a part of the displayimage to be displayed on display portion 161.

Mouse event generating portion 85 generates a mouse event in response toinput of positional information from operating portion 57. Further, eachtime mouse event generating portion 85 generates a mouse event, itoutputs the generated mouse event to gesture event determining portion83, and to browsing portion 61 via arbitration portion 53. When mouseevent generating portion 85 receives a plurality of pieces of positionalinformation simultaneously from operating portion 57, mouse eventgenerating portion 85 generates a plurality of mouse eventscorresponding respectively to the pieces of positional information. Inthe case where mouse event generating portion 85 generates one or moremouse events, it outputs the generated one or more mouse events togesture event determining portion 83 and browsing portion 61. Thepositional information indicates a position on the display surface ofdisplay portion 161, designated by an operation user.

The operations by which an operation user designates a position on thedisplay surface of display portion 161 include: a designation startingoperation by which a user designates an arbitrary position on thedisplay surface of display portion 161; a move operation by which theuser moves the designated position while designating (or, touching) thedisplay surface of display portion 161; and an exit operation by whichthe user terminates the designation performed on the display surface ofdisplay portion 161. When mouse event generating portion 85 receivespositional information from operating portion 57, mouse event generatingportion 85 determines whether the operation performed by the operationuser is the designation starting operation, the move operation, or theexit operation. As stated above, each time the positional information isoutput from touch panel 165, operating portion 57 outputs the positionalinformation. Therefore, mouse event generating portion 85 determineswhether the operation performed by the operation user is the designationstarting operation, the move operation, or the exit operation, on thebasis of the continuity of the pieces of positional information whichare input continuously over time.

When mouse event generating portion 85 receives positional informationfrom operating portion 57 after no positional information has beenreceived therefrom for a prescribed period of time, mouse eventgenerating portion 85 determines, for the received positionalinformation, that the operation user has performed the designationstarting operation. Then, mouse event generating portion 85 generates amouse event including the positional information and stateidentification information “Press” for identifying the designationstarting operation.

Following the detection of the mouse event whose state identificationinformation is “Press”, when mouse event generating portion 85 receivesone or more pieces of positional information continuously from operatingportion 57, mouse event generating portion 85 determines, for each ofthe received pieces of positional information, that the operation userhas performed the move operation. Then, for each of the one or morepieces of positional information input continuously from operatingportion 57, mouse event generating portion 85 generates a mouse eventincluding the positional information and state identificationinformation “Move” for identifying the move operation.

Following the detection of the mouse event whose state identificationinformation is “Move”, when mouse event generating portion 85 no longerreceives positional information from operating portion 57, mouse eventgenerating portion 85 determines, for the lastly received positionalinformation, that the operation user has performed the exit operation.Then, mouse event generating portion 85 generates a mouse eventincluding the lastly received positional information and stateidentification information “Release” for identifying the exit operation.

More specifically, mouse event generating portion 85 determines thestate of a mouse event on the basis of the positional information thatis input from operating portion 57 at a first time and the positionalinformation that is input from operating portion 57 at a second timewhich is a prescribed time after the first time. In the case wherepositional information is input at the first time but no positionalinformation indicating a position within the range of a prescribeddistance from the position specified by the positional information inputat the first time was input a prescribed time before the first time,then mouse event generating portion 85 determines the state of the mouseevent for the positional information input at the first time to be“Press”. In the case where the position specified by the positionalinformation input at the first time does not fall within the range of aprescribed distance from the position specified by the positionalinformation input at the second time, then mouse event generatingportion 85 determines the state of the mouse event for the positionalinformation input at the second time to be “Press”.

Further, in the case where the position specified by the positionalinformation input at the first time falls within the range of aprescribed distance from the position specified by the positionalinformation input at the second time, then mouse event generatingportion 85 determines the state of the mouse event for the positionalinformation input at the second time to be “Move”. Further, in the casewhere no positional information indicating a position within the rangeof a prescribed distance from the position specified by the positionalinformation input at the first time is input at the second time, thenmouse event generating portion 85 determines, at the second time, thestate of the mouse event for the positional information that was inputat the first time to be “Release”.

Gesture event determining portion 83 determines a gesture event on thebasis of a plurality of mouse events continuously input from mouse eventgenerating portion 85. In the case where mouse events are continuouslyinput one at a time from mouse event generating portion 85, gestureevent determining portion 83 specifies a second mouse event which isinput a prescribed time after the first mouse event was input. Then, inthe case where the distance between the position specified by the firstmouse event and the position specified by the second mouse event isequal to or greater than a prescribed distance, gesture eventdetermining portion 83 determines a gesture event, whereas when thedistance between the two positions is smaller than the prescribeddistance, gesture event determining portion 83 refrains from determininga gesture event. The prescribed time may be determined arbitrarily. Forexample, the prescribed time can be set to several times the interval ofdetection of positions by touch panel 165. The prescribed distance mayalso be determined arbitrarily, although it is preferably set to 40pixels, for example.

In the case where two or more mouse events are simultaneously input frommouse event generating portion 85, gesture event determining portion 83determines a gesture event on the basis of the two or more mouse events.Therefore, when a user inputs a multi-touch operation, gesture eventdetermining portion 83 determines a gesture event. Specifically, in thecase where a first mouse event and a second mouse event are inputsimultaneously from mouse event generating portion 85, gesture eventdetermining portion 83 determines a gesture event on the basis of afirst set of a plurality of mouse events which are continuously input inrelation to the first mouse event, and a second set of a plurality ofmouse events which are continuously input in relation to the secondmouse event. The mouse events included in the first set have theirpositional information identical to or continuous with one another.Here, being “continuous” means that two pieces of positional informationindicate positions adjacent to each other. The mouse events included inthe second set have their positional information identical to orcontinuous with one another. Therefore, gesture event determiningportion 83 refers to the positional information included in each of thesimultaneously input mouse events, to determine whether the mouse eventbelongs to the first set or the second set.

When gesture event determining portion 83 receives mouse eventscontinuously from mouse event generating portion 85, the stateidentification information of the firstly input mouse event is “Press”.Then, when gesture event determining portion 83 receives a mouse eventwhose state identification information is “Move”, gesture eventdetermining portion 83 calculates a distance between the two points, onthe basis of the positional information included in the mouse eventwhose state identification information is “Press” and the positionalinformation included in the mouse event whose state identificationinformation is “Move”, and detects a user's swipe operation if thedistance is a prescribed threshold value TH or greater, while it detectsa user's flick operation if the distance is smaller than the prescribedthreshold value TH. When a plurality of mouse events are input, a user'sswipe operation may be detected by earlier mouse events and a user'sflick operation may be detected by later mouse events. When gestureevent determining portion 83 detects a swipe operation or a flickoperation, it calculates a direction from the position specified by thepositional information included in the preceding mouse event toward theposition specified by the positional information included in thesucceeding mouse event, determines one of the left, right, top, and downdirections relative to the display surface of display portion 161 thatis closest to the calculated direction, as a parameter, and determines agesture event including the parameter. When gesture event determiningportion 83 detects a swipe operation, it determines a gesture eventwhich includes the gesture identification information “Swipe” foridentifying the gesture event and a parameter indicating a direction.When gesture event determining portion 83 detects a flick operation, itdetermines a gesture event which includes the gesture identificationinformation “Flick” and a parameter indicating a direction.

In the case where gesture event determining portion 83 receives twomouse events simultaneously from mouse event generating portion 85, thestate of each of the firstly input two mouse events is “Press”. In thiscase, gesture event determining portion 83 calculates, on the basis ofthe positional information included in the respective mouse events, adistance L1 between the two points and an angle R1 of a straight lineconnecting the two points. The angle of the straight line connecting thetwo points may be an angle between the straight line and a referenceline extending in a lateral direction of display portion 161. The anglebetween the reference line and the straight line connecting the twopoints is calculated clockwise from the reference line. Then, as twomouse events are subsequently input continuously, gesture eventdetermining portion 83 calculates, on the basis of the positionalinformation included in the subsequently input two mouse events, adistance L2 between the two points and an angle R2 of a straight lineconnecting the two points.

In the case where the angle R1 and the angle R2 are the same and thedistance L2 is different from the distance L1, then gesture eventdetermining portion 83 detects a pinch-out operation or a pinch-inoperation by a user. Here, the determination that the angle R1 and theangle R2 are the same means that the difference between the angle R1 andthe angle R2 is within a prescribed range. A certain margin of error isset for detecting the motions of the user fingers. If the distance L2 isgreater than the distance L1, gesture event determining portion 83detects a pinch-out operation. If the distance L2 is smaller than thedistance L1, gesture event determining portion 83 detects a pinch-inoperation. When detecting a pinch-out or pinch-in operation, gestureevent determining portion 83 calculates a scaling factor by dividing thedistance L2 by the distance L1, determines the calculated scaling factoras a parameter, and determines a gesture event including the parameter.When gesture event determining portion 83 detects a pinch-out operation,it determines a gesture event including the gesture identificationinformation “Pinch Out” and a parameter indicating a scaling factor.When gesture event determining portion 83 detects a pinch-in operation,it determines a gesture event including the gesture identificationinformation “Pinch In” and a parameter indicating a scaling factor.

In the case where the angle R1 and the angle R2 are different from eachother, gesture event determining portion 83 detects a rotation operationby a user. Here, the determination that the angle R1 and the angle R2are different from each other means that the difference between theangle R1 and the angle R2 is outside the prescribed range. A certainmargin of error is set for detecting the motions of the user fingers.When detecting a rotation operation, gesture event determining portion83 determines the direction of rotation by subtracting the angle R1 fromthe angle R2, determines the determined rotational direction as aparameter, and determines a gesture event including the parameter. Ifthe value obtained by subtracting the angle R1 from the angle R2 ispositive, the rotational direction is clockwise. If the value isnegative, the rotational direction is counterclockwise. Gesture eventdetermining portion 83 determines a gesture event including the gestureidentification information “Rotation” and a parameter indicating adirection of rotation.

When gesture event determining portion 83 determines a gesture event,gesture event determining portion 83 reads, from HDD 115, master table91 corresponding to the application identification information of thecurrent task which has been set by arbitration portion 53. Specifically,gesture event determining portion 83 outputs an operating commandinstructing to read master table 91 to operating portion 57, andacquires master table 91 that operating portion 57 reads by controllingHDD 115. In the following description, it is assumed that browsingprogram 61 is the current task.

In response to determination of a gesture event, gesture eventdetermining portion 83 specifies, from among a plurality of types ofprocesses determined in master table 91, a process that is associatedwith the gesture event in correspondence with the application programbeing set in the currently selected state by arbitration portion 53.Here, as browsing portion 61 has been set as the current task byarbitration portion 53, gesture event determining portion 83 specifiesthe process that is associated with the gesture event by master table 91for the browsing program, as shown in FIG. 6. More specifically, gestureevent determining portion 83 extracts, from master table 91, a masterrecord in which the gesture identification information included in thedetermined gesture event is set in the “gesture” field, and acquires theprocess identification information that is set in the “process” field inthe extracted master record. When the process identification informationis acquired, gesture event determining portion 83 outputs the determinedgesture event to arbitration portion 53. When no process identificationinformation is acquired, gesture event determining portion 83 refrainsfrom outputting the determined gesture event to arbitration portion 53.Further, in the case where gesture event determining portion 83continuously receives mouse events, one at a time, from mouse eventgenerating portion 85 but determines no gesture event, gesture eventdetermining portion 83 outputs a non-determination signal indicatingthat no gesture event is determined, to arbitration portion 53.

Gesture event determining portion 83 outputs, to arbitration portion 53,only the gesture event corresponding to the process that can be executedby browsing portion 61 which has been set in the currently selectedstate by arbitration portion 53. Arbitration portion 53 outputs thegesture event to browsing portion 61 which is the current task. Thismakes it possible to cause browsing portion 61, which executes thebrowsing program, to perform the process corresponding to the gestureevent which is the operation input to touch panel 165 and specified by aset of mouse events. In other words, at the stage of programdevelopment, a browsing program can be developed without any concern forthe gesture events which can be accepted by touch panel 165 in MFP 100.For example, it is possible to develop a browsing program which acceptsno rotation operation.

Arbitration portion 53 receives a mouse event from mouse eventgenerating portion 85, receives a gesture event from gesture eventdetermining portion 83, and outputs the mouse event and the gestureevent to the current task which is one of a plurality of tasks inapplication portion 51 for CPU 111 to execute application programs.Here, application portion 51 includes browsing portion 61 and secondaryexecuting portion 63, and secondary executing portion 63 is started bybrowsing portion 61. Therefore, arbitration portion 53 sets browsingportion 61 in the currently selected state. That is, browsing portion 61is the current task. When arbitration portion 53 receives a mouse eventfrom mouse event generating portion 85, arbitration portion 53 outputsthe mouse event to browsing portion 61. When arbitration portion 53receives a gesture event from gesture event determining portion 83,arbitration portion 53 outputs the gesture event to browsing portion 61.

Browsing portion 61 receives from arbitration portion 53 the mouse eventoutput from mouse event generating portion 85, and receives fromarbitration portion 53 the gesture event output from gesture eventdetermining portion 83. A gesture event is determined from a set of oneor more mouse events. Therefore, a gesture event and a set of one ormore mouse events which determines the gesture event are generatedsimultaneously in response to one single-touch operation or onemulti-touch operation which is input by an operation user.

Browsing portion 61 specifies a process to be performed, from among aplurality of types of processes determined by the browsing program, inaccordance with a single-touch or multi-touch operation according towhich a gesture event is determined, among the single-touch ormulti-touch operations that are input to touch panel 165 by an operationuser who operates MFP 100. Browsing portion 61 then executes thespecified process.

The plurality of types of processes performed by browsing portion 61include processes for changing an image displayed on display portion161. The processes for changing an image include, for example, a processfor changing the image being displayed to an image of other content. Inthe case where a part of an image is being displayed on display portion161, the processes for changing an image include: a scroll process and apage turning process for changing the area, in the image, to bedisplayed on display portion 161; an enlargement process for zooming inon the image; a reduction process for zooming out of the image; and arotation process for changing the direction of the image.

Further, secondary executing portion 63 performs a process in accordancewith a single-touch operation which is input into an embedded area andon the basis of which no gesture event is determined, among thesingle-touch operations that are input to touch panel 165 by theoperation user who operates MFP 100. Specifically, in the case wheresecondary executing portion 63 executes a second application programdifferent from the browsing program, secondary executing portion 63determines a process to be performed, from among a plurality of types ofprocesses which are determined by the second application programdifferent from the browsing program, and performs the specified process.In this case, the plurality of types of processes performed by secondaryexecuting portion 63 include processes for changing an image displayedon display portion 161. The processes for changing an image include, forexample, a process for changing the image being displayed to an image ofother content. In the case where a part of an image is being displayedon display portion 161, the processes for changing an image include: ascroll process and a page turning process for changing the area, in theimage, to be displayed on display portion 161; an enlargement processfor zooming in on the image; a reduction process for zooming out of theimage; and a rotation process for changing the direction of the image.

Further, in the case where secondary executing portion 63 causes aserver, which is an external computer connected via the network, toexecute the second application program different from the browsingprogram, secondary executing portion 63 transmits a processing requestincluding the mouse event corresponding to the single-touch operation,to the server.

Selecting portion 75 included in browsing portion 61 selects a mouseevent or a gesture event. When selecting portion 75 selects the mouseevent, it outputs the mouse event to instructing portion 77. Whenselecting portion 75 selects the gesture event, it outputs the gestureevent to determining portion 73.

In the case where selecting portion 75 receives a gesture event,selecting portion 75 has received, before the reception of the gestureevent, one or more mouse events on the basis of which the gesture eventhas been determined. Selecting portion 75 temporarily stores a mouseevent which is input via arbitration portion 53 from mouse eventgenerating portion 85, until a gesture event is input via arbitrationportion 53 from gesture event determining portion 83, or until a signalindicating that no gesture event is determined is input via arbitrationportion 53 from gesture event determining portion 83. Selecting portion75 may receive one mouse event at a time from arbitration portion 53, ormay receive two mouse events simultaneously therefrom. When a userinputs a single-touch operation, one mouse event is received. When auser inputs a multi-touch operation, a plurality of mouse events arereceived.

<In the Case of Single-Touch Operation>

When selecting portion 75 receives a mouse event from arbitrationportion 53, selecting portion 75 determines whether the positionindicated by the positional information included in the mouse eventfalls within an embedded area in the display image generated by displayimage generating portion 71. When the position indicated by thepositional information included in the mouse event falls within theembedded area and when a signal indicating that no gesture event isdetermined is received via arbitration portion 53 from gesture eventdetermining portion 83, then selecting portion 75 selects the mouseevent. When the position indicated by the positional informationincluded in the mouse event falls within the embedded area and when agesture event is received via arbitration portion 53 from gesture eventdetermining portion 83, if the gesture event is of a prescribed type,selecting portion 75 refrains from selecting the mouse event, whereas ifthe gesture event is not of the prescribed type, selecting portion 75selects the mouse event.

Further, when the position indicated by the positional informationincluded in the mouse event falls out of the embedded area, selectingportion 75 refrains from selecting the mouse event.

Further, when the position indicated by the positional informationincluded in the mouse event is outside the embedded area, selectingportion 75 selects a gesture event. When the position indicated by thepositional information included in the mouse event falls within theembedded area and when a gesture event is received via arbitrationportion 53 from gesture event determining portion 83, if the gestureevent is of a prescribed type, selecting portion 75 selects the gestureevent, whereas if the gesture event is not of the prescribed type,selecting portion 75 refrains from selecting the gesture event. Theprescribed types of gesture events are predetermined by the browsingprogram. In the present embodiment, the gesture events acceptable by thebrowsing program include, as shown in master table 91 in FIG. 6,“Swipe”, “Pinch Out”, “Pinch In”, “Rotation”, “Flick”, and “Double Tap”.The prescribed types of gesture events can be determined arbitrarilyfrom among the gesture events acceptable by the browsing program. Forexample, “Swipe”, “Pinch Out”, “Pinch In”, “Rotation”, and “Flick” canbe determined as the prescribed types of gesture events. In this case,“Double Tap” is not the gesture event of the prescribed type.

<In the Case of Multi-Touch Operation>

When selecting portion 75 receives a plurality of mouse eventssimultaneously from arbitration portion 53, selecting portion 75 selectsno mouse event. Further, when a plurality of mouse events are receivedsimultaneously, if a gesture event is received via arbitration portion53 from gesture event determining portion 83, selecting portion 75selects the gesture event, irrespective of the positions indicated bythe pieces of positional information included in the respective mouseevents. In other words, when the position indicated by the positionalinformation included in each mouse event falls out of the embedded area,selecting portion 75 refrains from selecting the mouse events.

Further, when the position indicated by the positional informationincluded in each mouse event is outside the embedded area, selectingportion 75 selects a gesture event. When the position indicated by thepositional information included in each mouse event falls within theembedded area and when a gesture event is received via arbitrationportion 53 from gesture event determining portion 83, if the gestureevent is of a prescribed type, selecting portion 75 selects the gestureevent, whereas if the gesture event is not of the prescribed type,selecting portion 75 refrains from selecting the gesture event.

When instructing portion 77 receives a mouse event from selectingportion 75, instructing portion 77 outputs the mouse event to secondaryexecuting portion 63. In the case where secondary executing portion 63causes a server, which is an external computer connected via thenetwork, to execute a second application program different from thebrowsing program, in response to input of the mouse event frominstructing portion 77, secondary executing portion 63 transmits aprocessing request including the mouse event to the server, and updatesthe embedded image with an image received from the server. Specifically,secondary executing portion 63 outputs to display control portion 87 anapplication command instructing to overwrite the VRAM included indisplay portion 161 with the image received from the server. This causesthe embedded area in the display image being stored in the VRAM indisplay portion 161, to be overwritten with the image received from theserver.

Further, in the case where secondary executing portion 63 executes thesecond application program different from the browsing program,secondary executing portion 63 specifies a process to be performed, fromamong a plurality of types of processes determined by the secondapplication program, on the basis of the received mouse event. Secondaryexecuting portion 63 then performs the specified process. For example,when the position specified by the positional information included inthe mouse event falls within an image of a button included in theembedded area, secondary executing portion 63 specifies a process thatis associated with the image of the button. For example, if the positionfalls within an image of a zoom-in button, secondary executing portion63 performs a process of zooming in on the image, and updates theembedded image with the zoomed image. Specifically, secondary executingportion 63 outputs to display control portion 87 an application commandinstructing to overwrite the VRAM included in display portion 161 withthe updated image. This causes the embedded area in the display imagebeing stored in the VRAM in display portion 161, to be overwritten withthe updated image.

When determining portion 73 receives a gesture event from selectingportion 75, determining portion 73 performs a process of the typecorresponding to the gesture event, among a plurality of types ofprocesses determined by the browsing program. Specifically, determiningportion 73 determines, from a display image, a display area to bedisplayed on display portion 161. Determining portion 73 outputs anapplication command for causing the determined display area to bedisplayed on display portion 161, to display control portion 87. Thiscauses an image of the area that has been determined by determiningportion 73 from within the display image stored in the VRAM in displayportion 161, to be displayed on display portion 161. For example, in thecase where the gesture event input from selecting portion 75 includesthe process identification information “Scroll”, an area within thedisplay image obtained by sliding the area that has been displayed ondisplay portion 161 to the left, right, up, or down, is displayed ondisplay portion 161.

A description will now be made of an exemplary flow of user operationsand processes performed by MFP 100 of the present embodiment, by takingas an example the case where MFP 100 downloads a Web page from a server300 connected to the Internet and displays a display image which is animage of the Web page.

FIG. 7 is a flowchart illustrating an example of the flow of theoperations by a user and processes performed by the MFP. Referring toFIG. 7, axes are shown, starting from the left, for the operations by auser, processes by operating portion 57, processes by secondaryexecuting portion 63, processes by browsing portion 61, and processes byserver 300, and time flows from the top downward at each axis.

First, when a user inputs an operation to start a browsing program,operating portion 57 accepts the operation. It is here assumed that aURL of a Web page stored in server 300 is designated by the user at thesame time. In accordance with the user operation, operating portion 57causes CPU 111 to execute the browsing program, whereby browsing portion61 which is the task for CPU 111 to execute the browsing program isstarted. Further, operating portion 57 outputs the URL input by theuser, to browsing portion 61.

Browsing portion 61 transmits a content request to server 300. Thecontent request is a request to server 300, specified by the URLreceived from operating portion 57, for transmission of the Web pagespecified by the URL. When server 300 receives the content request,server 300 returns the Web page specified by the URL, so that browsingportion 61 receives the Web page from server 300. Browsing portion 61analyzes the Web page, generates a display image corresponding to theWeb page, and outputs an instruction to store the display image into theVRAM and an instruction to display the display area, to operatingportion 57. In response thereto, operating portion 57 stores the displayimage into the VRAM included in display portion 161, and displays thedisplay area on display portion 161.

It is here assumed that the Web page includes a description of a commandto start a Java (registered trademark) program. Browsing portion 61 thuscauses CPU 111 to execute the Java (registered trademark) program, sothat secondary executing portion 63, which is the task for CPU 111 toexecute the Java (registered trademark) program, is started.

Here, it is also assumed that a process for causing server 300 totransmit an image of the content stored therein has been determined bythe Java (registered trademark) program. Secondary executing portion 63transmits to server 300 a processing request which is a request toperform the process of transmitting the image of the content. Inresponse thereto, server 300 returns the image of the content, so thatsecondary executing portion 63 receives the image of the content,transmitted from server 300, as an embedded image. Further, secondaryexecuting portion 63 stores the embedded image into the VRAM in displayportion 161, and outputs a display request to browsing portion 61. Inresponse thereto, operating portion 57 stores the display image into theVRAM in display portion 161. Furthermore, browsing portion 61 outputs tooperating portion 57 an instruction to display a display area. Operatingportion 57 displays on display portion 161 a display area from withinthe display image having the embedded image arranged in the embeddedarea. Therefore, the embedded image is displayed on display portion 161when the embedded area is included in the display area.

Thereafter, when the user inputs a single-touch operation or amulti-touch operation, operating portion 57 accepts the operation.Operation converting portion 55 generates one or more mouse eventscorresponding to the single-touch or multi-touch operation, anddetermines a gesture event on the basis of the one or more mouse events.The mouse event(s) and gesture event are input to browsing portion 61.

The processing performed by browsing portion 61 varies depending uponwhether the position indicated by the positional information included inany mouse event falls within the embedded area. In the case where theposition indicated by the positional information included in a mouseevent falls within the embedded area, the processing performed bybrowsing portion 61 further varies depending upon whether the gestureevent is of the prescribed type. In the case where the gesture event isnot of the prescribed type, browsing portion 61 outputs the mouse eventto secondary executing portion 63. Secondary executing portion 63transmits a processing request including the mouse event, to server 300.When server 300 receives the processing request, server 300 performs aprocess that has been determined by the second application program,which server 300 executes, in correspondence with the position in theembedded image specified by the mouse event included in the processingrequest, and returns the processed image of the content. For example, inthe case where the process of zooming in on an image has been determinedby the second application program, server 300 returns an image obtainedby zooming in on the image of the content. Secondary executing portion63 receives the image of the content transmitted from server 300, as theembedded image. Further, secondary executing portion 63 stores theembedded image into the VRAM included in display portion 161, andoutputs a display request to browsing portion 61. Correspondingly,operating portion 57 stores the display image into the VRAM included indisplay portion 161. Further, browsing portion 61 outputs to operatingportion 57 an instruction to display a display area. Operating portion57 displays on display portion 161 a display area from within thedisplay image having the embedded image arranged in the embedded area.Therefore, the embedded image, which has been enlarged by server 300, isdisplayed on display portion 161 when the embedded area is included inthe display area.

In the case where the position indicated by the positional informationincluded in the mouse event falls within the embedded area and when thegesture event is of the prescribed type, browsing portion 61 performs aprocess based on the gesture event. In this case, browsing portion 61refrains from outputting the mouse event to secondary executing portion63. The process based on the gesture event has been determined by thebrowsing program. For example, if the process based on the gesture eventis a downward scroll process, browsing portion 61 outputs to operatingportion 57 an instruction to display a display area obtained by movingthe display area downward in the display image. Operating portion 57displays on display portion 161 the display area in the display image.Therefore, the display area which has been moved in the display image isdisplayed on display portion 161.

On the other hand, in the case where the position indicated by thepositional information included in the mouse event falls out of theembedded area, browsing portion 61 performs a process based on thegesture event. In this case, browsing portion 61 refrains fromoutputting the mouse event to secondary executing portion 63.

FIG. 8 is a flowchart illustrating an example of the flow of a gestureevent determining process. The gesture event determining process iscarried out by CPU 111 included in MFP 100 as CPU 111 executes anoperation accepting program stored in ROM 113, HDD 115, or CD-ROM 118.Referring to FIG. 8, CPU 111 determines whether a mouse event has beendetected (step S01). If a user designates a position on touch panel 165,a mouse event is detected on the basis of the position detected by touchpanel 165. CPU 111 is in a standby mode until a mouse even is detected(NO in step S01), and once a mouse event is detected (YES in step S01),the process proceeds to step S02.

In other words, the gesture event determining process is executed on thecondition that a user designates touch panel 165 and a mouse event isdetected.

In step S02, CPU 111 determines whether a single mouse event has beendetected. If a single mouse event has been detected, the processproceeds to step S03. If more than one mouse event has been detected,the process proceeds to step S17.

In step S03, the process branches in accordance with the state of themouse event. If the state of the mouse event is “Press”, the processproceeds to step S04. If the state of the mouse event is “Move”, theprocess proceeds to step S09. If the state of the mouse event is“Release”, the process proceeds to step S05. In step S04, CPU 111 startsa timer T, and the process returns to step S01. Timer T counts timeelapsed from the detection of a mouse event whose state is “Press”.

In step S09, CPU 111 determines whether a value of timer T is athreshold value TT or less. If so, the process proceeds to step S10;otherwise, the process proceeds to step S11. In step S10, CPU 111determines whether spacing is a threshold value TL or greater. If so,the process proceeds to step S12; otherwise, the process proceeds tostep S11. The spacing is a distance between the position specified bythe positional information included in the firstly detected mouse event,i.e. the mouse event whose state is “Press”, and the position specifiedby the positional information included in the mouse event detected instep S01 most recently. In step S11, CPU 111 output a non-determinationsignal indicating that no gesture event is determined, and the processis terminated. That is, in the case where no mouse event indicating aposition at a distance of threshold value TL or greater from theposition indicated by the mouse event whose state is “Press” has beendetected within the time of threshold value TT from the detection of themouse event whose state is “Press”, the process is terminated with nogesture event determined. This is to prevent any gesture event frombeing determined in the case where the position designated by a user isnot moved within a lapse of time corresponding to the threshold valueTT. Alternatively, it may be configured to determine a gesture eventthat a user designates the same position for a prescribed time.

The process proceeds to step S12 when the state of the mouse event is“Move”. This means that another mouse event has been input immediatelybefore the mouse event. The state of the other mouse event may be“Press” or “Move”. In step S12, CPU 111 determines a designateddirection. Specifically, CPU 111 determines the direction from theposition specified by the positional information included in the othermouse event input immediately before, to the position specified by thepositional information included in the mouse event detected in step S01.The direction determined here is one of the left, right, up, and downdirections relative to the display surface of display portion 161 thatis closest to the calculated direction.

In the following step S13, CPU 111 calculates a designated distance L.Specifically, CPU 111 calculates, as the designated distance L, adistance between the position specified by the positional informationincluded in the other mouse event input immediately before, and theposition specified by the positional information included in the mouseevent detected in step S01. CPU 111 then determines whether thedesignated distance L is not smaller than a threshold value TH (stepS14). If the designated distance L is equal to or greater than thethreshold value TH, the process proceeds to step S15; otherwise, theprocess proceeds to step S16.

In step S15, CPU 111 determines the gesture event to be “Swipe”, and theprocess proceeds to step S27. In step S16, CPU 111 determines thegesture event to be “Flick”, and the process proceeds to step S27.

The process proceeds to step S05 when the state of the mouse eventdetected in step S01 is “Release”, in which case another mouse event hasbeen input immediately before the mouse event. The state of the othermouse event may be “Press” or “Move”. In step S05, the process branchesin accordance with the state of the other mouse event input immediatelybefore. If the state of the other mouse event input immediately beforeis “Press”, the process proceeds to step S06; otherwise, the processreturns to step S01.

In step S06, CPU 111 determines whether RAM 114 stores the samepositional information as the one included in the mouse event detectedin step S01. If so, the process proceeds to step S07; otherwise, theprocess proceeds to step S08. Here, the determination that the pieces ofpositional information are the same means that the distance between thetwo positions specified by the respective pieces of positionalinformation is within a prescribed length. A certain margin of error isset for detecting the user operations of designating the same position.

In step S08, CPU 111 temporarily stores the positional informationincluded in the mouse event detected in step S01, into RAM 114, and theprocess proceeds to step S29. The process proceeds to step S08 when thestate of the mouse event input immediately before is “Press” and thestate of the mouse event input subsequently is “Release”, and when thesame positional information has not been stored in RAM 114. In otherwords, the process proceeds to step S08 when a user inputs a first tapoperation.

In step S07, CPU 111 determines the gesture event to be “Double Tap”,and the process proceeds to step S27. The process proceeds to step S07when the state of the mouse event input immediately before is “Press”and the state of the mouse event detected in step S01 is “Release”, andwhen the same positional information has been stored in RAM 114. Inother words, the process proceeds to step S07 when a user, following thefirst tap operation, taps the same position as the firstly tappedposition.

The process proceeds to step S17 in the case where two mouse events aredetected. In step S17, CPU 111 determines whether the state of at leastone of the two mouse events is “Move”. If so, the process proceeds tostep S18; otherwise, the process returns to step S01.

In step S18, CPU 111 calculates an angle. Specifically, CPU 111calculates the angle between the straight line connecting the twopositions specified by the respective pieces of positional informationincluded in the two mouse events and a prescribed reference side on thedisplay surface of display portion 161. In the following step S19, CPU111 determines whether the angle calculated in step S18 has beenchanged. It is determined that the angle has been changed if thedifference between the angle calculated in step S18 and the anglecalculated on the basis of another set of two mouse events inputimmediately before is a prescribed value or greater. If the angle hasbeen changed, the process proceeds to step S20; otherwise, the processproceeds to step S21. In step S20, CPU 111 determines the gesture eventto be “Rotation”, and the process proceeds to step S27.

In step S21, CPU 111 calculates a distance. Specifically, CPU 111calculates the distance between the two positions specified by therespective pieces of positional information included in the two mouseevents. In the following step S22, CPU 111 determines whether thedistance calculated in step S21 has been changed. It is determined thatthe distance has been changed if the difference between the distancecalculated in step S21 and the distance calculated on the basis of theother set of two mouse events input immediately before is a prescribedvalue or greater. If the distance has been changed, the process proceedsto step S23; otherwise, the process returns to step S01.

In step S23, CPU 111 determines a scaling factor. Specifically, CPU 111determines the value obtained by dividing the distance calculated instep S21 by the distance calculated on the basis of the other set of twomouse events input immediately before, as the scaling factor. In thefollowing step S24, CPU 111 determines whether the scaling factor isgreater than 1. If so, the process proceeds to step S25; otherwise, theprocess proceeds to step S26.

In step S25, CPU 111 determines the gesture event to be “Pinch Out”, andthe process proceeds to step S27. In step S26, CPU 111 determines thegesture event to be “Pinch In”, and the process proceeds to step S27.

In step S27, CPU 111 determines whether a process corresponding to thedetermined gesture event has been defined. Specifically, CPU 111determines whether the plurality of types of processes that have beendetermined in master table 91 stored in HDD 115 include a process thatis associated with the relevant gesture event in correspondence with theapplication program being set in the currently selected state. If thereis a process associated with the determined gesture event in mastertable 91, CPU 111 determines that a process corresponding to thedetermined gesture event has been defined for the application programset in the currently selected state. If a process corresponding to thedetermined gesture event has been defined, the process proceeds to stepS28; otherwise, the process proceeds to step S29.

In step S28, CPU 111 outputs the determined gesture event, and theprocess is terminated. In step S29, CPU 111 outputs a non-determinationsignal, and the process is terminated.

FIG. 9 is a flowchart illustrating an example of the flow of a browsingprocess. The browsing process is carried out by CPU 111 included in MFP100 as CPU 111 executes a browsing program stored in ROM 113, HDD 115,or CD-ROM 118. Referring to FIG. 9, CPU 111 acquires content (step S31).Specifically, CPU 111 downloads a Web page as the content, from a serverconnected to the Internet. The Web page is identified by a URLdesignated by an operation user who operates MFP 100. Alternatively, theWeb page may be identified by a predetermined URL.

In step S32, CPU 111 determines whether the acquired Web page includes adescription of an embedded image. The description of an embedded imagerefers to a description for displaying an embedded image that isgenerated as a second application program different from the browsingprogram is executed. If the Web page acquired from the server includes adescription of an embedded image, the process proceeds to step S33;otherwise, the process proceeds to step S35. In step S35, CPU 111generates a display image in accordance with the description of the Webpage, and the process proceeds to step S36. Specifically, CPU 111 storesthe display image into the VRAM included in display portion 161.

In step S33, CPU 111 generates a display image including an embeddedarea. Specifically, CPU 111 stores the display image including theembedded area, into the VRAM included in display portion 161. Thiscauses the display image generated by the task of executing the browsingprogram to be stored in the VRAM in display portion 161. It is notedthat in the embedded area in the display image, nothing is stored by thetask of executing the browsing program.

In step S34, CPU 111 starts an application program, and the processproceeds to step S36. In accordance with the description of the Webpage, CPU 111 or the server is caused to execute a second applicationprogram different from the browsing program, and the task of executingthe second application program is caused to generate an embedded image.Specifically, in the case where the second application program isexecuted by the server in accordance with the JavaScript (registeredtrademark) described in the Web page, CPU 111 transmits a processingrequest for requesting the generation of content to the server, andstores the image of the content returned from the server into theembedded area in the display image stored in the VRAM in display portion161. As a result, in the embedded area in the display image which isstored in the VRAM included in display portion 161, the image that theserver has generated by executing the second application program isstored as the embedded image. In the case where the Web page includes aninstruction to start a second application program, such as a reproducingprogram, which can be executed by CPU 111, CPU 111 is caused to executethe second application program. In this case, the image generated by thetask for CPU 111 to execute the second application program is storedinto the embedded area in the display image stored in the VRAM indisplay portion 161. As a result, in the embedded area in the displayimage which is stored in the VRAM included in display portion 161, theembedded image that the task for CPU 111 to execute the secondapplication program has generated is stored.

In this manner, the task of executing the browsing program rewrites thedisplay image, and the embedded area is rewritten with the imagegenerated by the task for CPU 111 or the server to execute the secondapplication program. In other words, the embedded area can be updatedwith the image generated by the task for the server or CPU 111 toexecute the second application program, independently of the task ofexecuting the browsing program.

In step S36, CPU 111 displays the display image. Specifically, CPU 111displays the display image stored in the VRAM included in displayportion 161, on display portion 161. At this time, CPU 111 may display apart of the display image, or it may display the entire display image.

In step S37, CPU 111 determines whether one or more mouse events havebeen input. If an operation user uses his/her finger(s) to designate oneor more positions on the display surface of display portion 161, thedesignated position(s) is/are detected by touch panel 165, and one ormore mouse events each including the corresponding positionalinformation are input. If one or more mouse events have been input (YESin step S37), the process proceeds to step S38; otherwise, the processproceeds to step S50.

In step S38, CPU 111 determines whether a single mouse event has beeninput. If a single mouse event has been input, the process proceeds tostep S39. If more than one mouse event has been input, the processproceeds to step S48. In step S39, CPU 111 determines whether the mouseevent is an operation performed on the embedded area. If the positionspecified by the positional information included in the mouse eventfalls within the embedded area in the display image, CPU 111 determinesthat the mouse event is an operation performed on the embedded area. Ifso, the process proceeds to step S40; otherwise, the process proceeds tostep S48.

In step S40, CPU 111 determines whether a non-determination signal hasbeen input. In the above-described gesture event determining process,when it is determined that no gesture event is determined on the basisof the mouse event which has been input in step S37, a non-determinationsignal is input. If a non-determination signal has been input, theprocess proceeds to step S41; otherwise, the process proceeds to stepS43. In step S41, CPU 111 outputs the mouse event to the task ofexecuting the second application program which has been started in stepS34, and the process proceeds to step S42. The mouse event to be outputhere includes the mouse event which is input in step S37, and the mouseevent which is temporarily stored in step S47 which will be describedlater. In the case of outputting a plurality of mouse events, CPU 111outputs the mouse events in the same order as they were input.Specifically, in the case where the second application program isexecuted by the server in accordance with the JavaScript (registeredtrademark) described in the Web page, CPU 111 transmits a processingrequest including the mouse event to the server, receives an image ofcontent returned from the server, and stores the received image into theembedded area in the display image being stored in the VRAM included indisplay portion 161. In the case where the Web page includes aninstruction to start the second application program, such as areproducing program, executable by CPU 111, the task of executing thesecond application program has been formed in CPU 111. Therefore, themouse event is output to the task for CPU 111 to execute the secondapplication program to cause the task to perform the process based onthe mouse event, and an image of content that the task for CPU 111 toexecute the second application program has generated is stored into theembedded area in the display image being stored in the VRAM included indisplay portion 161.

When the task for the server or CPU 111 to execute the secondapplication program receives the mouse event, the task performs aprocess specified by the mouse event, among a plurality of processesdetermined by the second application program. Specifically, the taskperforms the process, among the plurality of processes determined by thesecond application program, that is assigned to the position specifiedby the positional information included in the mouse event. For example,when the embedded image includes a button assigned with an enlargementprocess and when the position specified by the positional informationincluded in the mouse event falls within the button assigned with theenlargement process, then the task performs the enlargement process. Thetask then updates the embedded area in the display image stored in theVRAM in display portion 161, with an embedded image obtained by theenlargement process. As a result, in the display image being displayedon display portion 161, only the image in the embedded area is enlargedfor display.

In the following step S42, CPU 111 erases the mouse event which has beentemporarily stored in step S47, and the process proceeds to step S50.This is for preventing the same mouse event from being output twice tothe task of executing the second application program. On the other hand,in step S43, CPU 111 determines whether a gesture event has beendetermined. Specifically, CPU 111 determines whether the gesture eventdetermining process shown in FIG. 8 has been performed and a gestureevent has been determined on the basis of the mouse event which has beeninput in step S37. If the gesture event has been determined, the processproceeds to step S44; otherwise, the process proceeds to step S47. Instep S47, CPU 111 temporarily stores the mouse event which has beeninput in step S37, and the process proceeds to step S50.

In step S44, CPU 111 determines whether the gesture event that has beendetermined on the basis of the mouse event input in step S37 is of aprescribed type. If so, the process proceeds to step S45; otherwise, theprocess proceeds to step S41. The prescribed types of gesture eventshave been predetermined by the browsing program.

In step S45, CPU 111 performs a process corresponding to the gestureevent, and the process proceeds to step S46. CPU 111 performs one of theplurality of processes determined by the browsing program that isspecified by the gesture event. Specifically, if the gesture event is“Flick”, CPU 111 moves the area to be displayed on display portion 161,in the display image, in the direction determined by the gesture event.More specifically, CPU 111 moves the area to be displayed on displayportion 161, in the display image stored in the VRAM included in displayportion 161, in the direction determined by the gesture event, anddisplays the moved area on display portion 161. This causes the area, inthe display image, being displayed on display portion 161 to be changed.In step S46, CPU 111 erases the mouse event which has been temporarilystored in step S47, as in step S42, and the process proceeds to stepS50.

On the other hand, the process proceeds to step S48 when a plurality ofmouse events are input in step S37. In step S48, CPU 111 determineswhether a gesture event has been determined. CPU 111 determines whetherthe gesture event determining process shown in FIG. 8 has been performedand a gesture event has been determined on the basis of the mouse eventsinput in step S37. If the gesture event has been determined, the processproceeds to step S49; otherwise, the process proceeds to step S50. Instep S49, CPU 111 performs a process corresponding to the gesture event,and the process proceeds to step S50.

In step S50, CPU 111 determines whether it has accepted an instructionto terminate the display of the content. For example, if one of thekeys, included in hard key portion 167, that is assigned with aninstruction to terminate the browsing program has been designated, CPU111 accepts the instruction to terminate the display of the content. Ifso, the process is terminated; otherwise, the process returns to stepS37.

FIG. 10 shows an example of a display image. The display image shown inFIG. 10 corresponds to a Web page which has been downloaded from aserver connected to the Internet. It is here assumed that the Web pagecontains an introduction to a shop selling products. Referring to FIG.10, a display image 300 includes two embedded areas 310 and 311 in eachof which an embedded image generated by a second application programdifferent from the browsing program is arranged in the Web page. Theembedded images to be arranged in the embedded areas 310, 311 aregenerated as the server connected to the Internet executes the secondapplication program installed into the server.

In embedded area 310, an image of a map showing the location of the shopis arranged as the embedded image. In embedded area 311, an imageshowing the list of products sold by the shop is arranged as theembedded image. Areas 301 and 303 delimited by dotted lines each show adisplay area, in display image 300, to be displayed on display portion161. It is noted that the dotted lines delimiting areas 301 and 303 areshown for illustration purposes only; the dotted lines do not exist indisplay image 300.

FIG. 11 shows a first example of a display area in display image 300.Referring to FIG. 11, a display area 331 is an area indicated by area301 in display image 300 shown in FIG. 10. Display area 331 includesembedded area 310 arranged in display image 300. Embedded area 310includes buttons 313 having the left, right, up, and down arrowsrespectively displayed thereon, and buttons 315 having the signs “+” and“−” respectively displayed thereon. Each button in buttons 313 and 315is assigned with a corresponding process by the second applicationprogram which has been installed into the server. When a user useshis/her finger to designate the “+” button in buttons 315, a mouse eventincluding the positional information indicating a position fallingwithin the range where the “+” button is displayed is generated. Theflows of the gesture event determining process shown in FIG. 8 and thebrowsing process shown in FIG. 9 corresponding to this case will bedescribed specifically. In the gesture event determining process shownin FIG. 8, when the user designates the “+” button in buttons 315 withhis/her finger, “Release” is detected as the state of the mouse event instep S03, and the process proceeds to step S06. It is then determinedthat the same position has not been stored, so that steps S08 and S29are performed. Therefore, a non-determination signal is output, with nogesture event detected. Further, in the browsing process shown in FIG.9, when the user designates the “+” button in buttons 315 with his/herfinger, it is determined in step S37 that a mouse event has been input,and the process proceeds to step S38. In the user operation ofdesignating the “+” button in buttons 315 with his/her finger, a singleposition falling within the embedded area is designated, and therefore,the process proceeds to step S40. In the gesture event determiningprocess shown in FIG. 8, when the user designates the “+” button inbuttons 315 with his/her finger, a non-determination signal is output,and therefore, in the browsing process, step S41 is performed in which amouse event is output. Consequently, the mouse event is transmitted tothe server which is responsible for generating the image arranged inembedded area 310, and the server performs the process of zooming in onthe image. Then, embedded area 310 is updated with the image that hasundergone the enlargement process by the server.

FIG. 12 shows a second example of the display area in display image 300.Referring to FIG. 12, a display area 331 is an area indicated by area301 in display image 300 shown in FIG. 10. Display area 331 includesembedded area 310 arranged in display image 300. The embedded imagedisplayed in embedded area 310 is an enlarged version of the imagearranged in embedded area 310 shown in FIG. 11. Except for the embeddedarea 310, display area 331 in display image 300 shown in FIG. 12 isidentical to display area 331 shown in FIG. 11.

Further, when the operation user touches with his/her finger anyposition on display area 331 in display image 300 shown in FIG. 12 toinput a swipe operation, a gesture event is generated. The flows of thegesture event determining process shown in FIG. 8 and the browsingprocess shown in FIG. 9 corresponding to this case will be describedspecifically. In the gesture event determining process shown in FIG. 8,when the user inputs a swipe operation, “Move” is detected as the stateof the mouse event in step S03, and the process proceeds to step S09. Inthe swipe operation, the designated position is moved and the distanceof movement per unit time is not less than the threshold value T11, sothat the process proceeds to step S15. In this manner, the gesture eventwith the gesture identification information “Swipe” is determined.Further, in the browsing process shown in FIG. 9, when the user inputsthe swipe operation, it is determined in step S37 that a mouse event hasbeen input, and the process proceeds to step S38. The swipe operation isan operation of moving the position designated with one finger, so thatthe process proceeds to step S40 or S48. In the case where the processproceeds to step S40, or in other words, in the case where the swipedposition falls within the embedded area, the process proceeds to stepS45, because a gesture event with the gesture identification information“Swipe” is determined in the gesture event determining process shown inFIG. 8. On the other hand, in the case where the process proceeds tostep S48, or in other words, in the case where the swiped position isoutside the embedded area, the process proceeds to step S49, because agesture event with the gesture identification information “Swipe” isdetermined in the gesture event determining process shown in FIG. 8. Assuch, the position touched by the user's finger may be anywhere withindisplay area 331; it may fall within the embedded area 310. Although amouse event is generated together with the gesture event, the mouseevent is discarded, because the gesture event corresponds to a processof a prescribed type which is predetermined by the browsing program. Thebrowsing program has determined, for the gesture event with the gestureidentification information “Swipe”, the page turning process of changingthe display area in the display image. It is here assumed that the pageturning process is performed so that the display area in display image300 is changed from area 301 to area 303.

FIG. 13 shows a third example of the display area in display image 300.Shown in FIG. 13 is a display area 341, included in display image 300,which is displayed on display portion 161 after a swipe operation isinput in the state where display area 331 shown in FIG. 12 is beingdisplayed on display portion 161. Display area 341 is an area indicatedby area 303 in display image 300 shown in FIG. 10. Display area 341includes embedded area 311 arranged in display image 300. The embeddedimage displayed in embedded area 311 is the same as the one arranged inembedded area 311 shown in FIG. 10.

As described above, MFP 100 according to the present embodimentfunctions as a data processing apparatus. When one or more positionsdesignated by a user are detected by touch panel 165, MFP 100 generatesone or more mouse events each including a piece of positionalinformation indicating the corresponding position, and determines one ofa plurality of gesture events on the basis of a set of the one or moremouse events generated. On the other hand, in the case where a Web pagereceived from the outside includes a description for displaying anembedded image generated as an application program different from abrowsing program is executed, MFP 100 generates a display image havingthe embedded image arranged in an embedded area determined by the Webpage. When the display image includes the embedded area, in the casewhere the position indicated by one of the one or more mouse events onthe basis of which the gesture event has been determined falls withinthe embedded area, MFP 100 selects either the gesture event or the setof the one or more mouse events on the basis of which the gesture eventhas been determined. Therefore, MFP 100 is able to detect onedesignation operation made by a user on the display surface of displayportion 161 as one or more mouse events and as a gesture event, and isable to accept either the detected mouse event(s) or the detectedgesture event which is suitable for a respective one of the browsingprogram and the application program different from the browsing program.Accordingly, even in the case where a Web page includes a descriptionfor executing an application program different from the browsingprogram, MFP 100 can execute the application program appropriately.

Further, in the case where MFP 100 detects one designation operationmade by the user on the display surface of display portion 161 as one ormore mouse events and as a gesture event, when the gesture event is of aprescribed type, MFP 100 causes the gesture event to correspond to thebrowsing program, whereas when the gesture event is not of theprescribed type, MFP 100 causes the one or more mouse events tocorrespond to the application program different from the browsingprogram. Accordingly, MFP 100 is able to accept a user operation as anoperation corresponding to either the browsing program or theapplication program. Furthermore, in the case where the browsing programcorresponds to the gesture event and the application program correspondsto the mouse event, MFP 100 can execute the application programappropriately.

Further, in the case where the spacing between a first positionindicated by a first mouse event and a second position indicated by asecond mouse event generated within a prescribed time from thegeneration of the first mouse event is smaller than a prescribeddistance, MFP 100 refrains from determining a gesture event.Accordingly, MFP 100 is able to detect a user operation of moving thedesignated position on touch panel 165 as a gesture event.

Furthermore, MFP 100 holds (temporarily stores) the generated one ormore mouse events until a gesture event is determined or until it isdecided not to determine a gesture event. When the set of the one ormore mouse events is selected, MFP 100 outputs the one or more mouseevents put on hold, to the task of executing the application program. Assuch, MFP 100 holds any mouse event generated on the basis of a useroperation which has been input before a gesture event which is not of aprescribed type is determined or before it is decided not to determine agesture event. This ensures effectiveness of the user operation; it isunnecessary for the user to input the same operation again.

Furthermore, MFP 100 erases the one or more mouse events put on hold,after they are output to the task of executing the application program.In the case where the gesture event is selected, MFP 100 erases themouse event(s) at the time when the gesture event is selected. This canprevent a same mouse event from being output more than once.

In the case where the mouse event is selected, MFP 100 outputs the mouseevent to the task of executing the application program, to cause thetask to update the embedded image displayed in the embedded area. Thiscan cause the task of executing the application program to execute aprocess in accordance with the mouse event.

Further, in the case where the gesture event is selected, MFP 100executes a process of the type corresponding to the selected gestureevent, among a plurality of types of processes executable by thebrowsing program. This can cause the task of executing the browsingprogram to perform the process of determining the display area, in thedisplay image, to be displayed on display portion 161. It is possible tocause the task of executing the browsing program to perform a process inaccordance with the gesture event.

Furthermore, at the stage when the browsing program is installed, MFP100 stores into HDD 115 master table 91 in which a plurality of types ofprocesses determined by the browsing program are associated withcorresponding ones of gesture events. This makes it possible to detectonly the gesture events corresponding respectively to the plurality oftypes of processes determined by the browsing program.

While MFP 100 has been described as an example of the data processingapparatus in the above embodiment, the present invention may of coursebe understood as an operation accepting method for causing CPU 111controlling MFP 100 to perform the processing shown in FIGS. 8 and 9, oras a browsing program for causing CPU 111 to perform the operationaccepting method.

Although the present invention has been described and illustrated indetail, it is clearly understood that the same is by way of illustrationand example only and is not to be taken by way of limitation, the spiritand scope of the present invention being limited only by the terms ofthe appended claims.

What is claimed is:
 1. A data processing apparatus including a displayportion capable of displaying an image, a position detecting portioncapable of detecting one or more positions designated by a user on adisplay surface of said display portion, and a control portion, saidcontrol portion comprising: a browsing portion to execute a browsingprogram, externally acquire content described in a markup language, anddisplay on said display portion at least a part of a display imagecorresponding to said acquired content, and further operable, inresponse to input of one of a plurality of gesture events correspondingrespectively to a plurality of types of processes determined by saidbrowsing program, to perform the one of said plurality of types ofprocesses that corresponds to said input gesture event; a mouse eventgenerating portion operable, in response to detection of one or morepositions by said position detecting potion, to generate one or moremouse events each including a piece of positional information indicatingthe corresponding one of said one or more positions; and a gesture eventdetermining portion to determine one of the plurality of gesture eventson the basis of a set of one or more mouse events generated by saidmouse event generating portion; said browsing portion including adisplay image generating portion operable, in the case where saidcontent includes a description for displaying an embedded image to begenerated as an application program different from said browsing programis executed, to generate a display image in which the embedded imagegenerated as said application program has been executed is arranged inan embedded area determined by said content, and a selecting portionoperable, in the case where the position indicated by one of the one ormore mouse events on the basis of which the gesture event has beendetermined by said gesture event determining portion falls within saidembedded area, to select either the gesture event determined by saidgesture event determining portion or the set of the one or more mouseevents on the basis of which the gesture event has been determined, onthe basis of the type of said determined gesture event.
 2. The dataprocessing apparatus according to claim 1, wherein said selectingportion includes a mouse event selecting portion operable, in the casewhere the gesture event determined by said gesture event determiningportion is not of a prescribed type, to select the one or more mouseevents on the basis of which the gesture event has been determined bysaid gesture event determining portion, and a gesture event selectingportion operable, in the case where the gesture event determined by saidgesture event determining portion is of said prescribed type, to selectthe gesture event determined by said gesture event determining portion.3. The data processing apparatus according to claim 1, wherein saidgesture event determining portion includes a deciding portion to decidenot to determine a gesture event in the case where a spacing between afirst position indicated by a first mouse event generated by said mouseevent generating portion and a second position indicated by a secondmouse event generated within a prescribed time from the generation ofsaid first mouse event is less than a prescribed distance.
 4. The dataprocessing apparatus according to claim 3, wherein said browsing portionfurther includes a holding portion to hold a set of one or more mouseevents generated by said mouse event generating portion until a gestureevent is determined on the basis of the set by said gesture eventdetermining portion or until said deciding portion decides not todetermine said gesture event, and an instructing portion operable, inthe case where said set of one or more mouse events is selected by saidselecting portion, to output the one or more mouse events included insaid held set to the task of executing said application program to causethe task to update the embedded image displayed in said embedded area.5. The data processing apparatus according to claim 4, wherein saidbrowsing portion further includes an erasing portion operable, in thecase where said set of one or more mouse events is selected by saidselecting portion, to erase the one or more mouse events included insaid held set after the one or more mouse events included in said heldset are output to the task of executing said application program by saidinstructing portion, and operable, in the case where the gesture eventis selected by said selecting portion, to erase the one or more mouseevents included in said held set at the time when the gesture event isselected by said selecting portion.
 6. The data processing apparatusaccording to claim 1, wherein said browsing portion further includes adetermining portion operable, in the case where said gesture event isselected, to perform one of said plurality of types of processes thatcorresponds to said selected gesture event to thereby determine adisplay area, in said display image, to be displayed on said displayportion.
 7. The data processing apparatus according to claim 1, whereinsaid control portion further includes an association portion operable,at the stage when said browsing program is installed, to associate eachof said plurality of types of processes determined by said browsingprogram with one of a plurality of gesture events in correspondence withsaid browsing program, and said gesture event determining portiondetermines said gesture event on the condition that the gesture eventcorresponding to the set of one or more mouse events generated by saidmouse event generating portion is associated with one of said pluralityof types of processes by said association portion in correspondence withsaid browsing program.
 8. An operation accepting method performed by adata processing apparatus, the data processing apparatus including adisplay portion capable of displaying an image and a position detectingportion capable of detecting one or more positions designated by a useron a display surface of said display portion, the method comprising: abrowsing step of executing a browsing program, externally acquiringcontent described in a markup language, and displaying on said displayportion at least a part of a display image corresponding to saidacquired content, and further, in response to input of one of aplurality of gesture events corresponding respectively to a plurality oftypes of processes determined by said browsing program, performing theone of said plurality of types of processes that corresponds to saidinput gesture event; a mouse event generating step of, in response todetection of one or more positions by said position detecting potion,generating one or more mouse events each including a piece of positionalinformation indicating the corresponding one of said one or morepositions; and a gesture event determining step of determining one ofthe plurality of gesture events on the basis of a set of one or moremouse events generated in said mouse event generating step; saidbrowsing step including a display image generating step of, in the casewhere said content includes a description for displaying an embeddedimage which is generated as an application program different from saidbrowsing program is executed, generating a display image in which theembedded image generated as said application program has been executedis arranged in an embedded area determined by said content, and aselecting step of, in the case where the position indicated by one ofthe one or more mouse events on the basis of which the gesture event hasbeen determined in said gesture event determining step falls within saidembedded area, selecting either the gesture event determined in saidgesture event determining step or the set of the one or more mouseevents on the basis of which the gesture event has been determined, onthe basis of the type of said determined gesture event.
 9. The operationaccepting method according to claim 8, wherein said selecting stepincludes a mouse event selecting step of, in the case where the gestureevent determined in said gesture event determining step is not of aprescribed type, selecting the one or more mouse events on the basis ofwhich the gesture event has been determined in said gesture eventdetermining step, and a gesture event selecting step of, in the casewhere the gesture event determined in said gesture event determiningstep is of said prescribed type, selecting the gesture event determinedin said gesture event determining step.
 10. The operation acceptingmethod according to claim 8, wherein said gesture event determining stepincludes a deciding step of deciding not to determine a gesture event inthe case where a spacing between a first position indicated by a firstmouse event generated in said mouse event generating step and a secondposition indicated by a second mouse event generated within a prescribedtime from the generation of said first mouse event is less than aprescribed distance.
 11. The operation accepting method according toclaim 10, wherein said browsing step further includes a holding step ofholding a set of one or more mouse events generated in said mouse eventgenerating step until a gesture event is determined on the basis of theset in said gesture event determining step or until it is decided not todetermine said gesture event in said deciding step, and an instructingstep of, in the case where said set of one or more mouse events isselected in said selecting step, outputting the one or more mouse eventsincluded in said held set to the task of executing said applicationprogram to thereby cause the task to update the embedded image displayedin said embedded area.
 12. The operation accepting method according toclaim 11, wherein said browsing step further includes an erasing stepof, in the case where said set of one or more mouse events is selectedin said selecting step, erasing the one or more mouse events included insaid held set after the one or more mouse events included in said heldset are output to the task of executing said application program in saidinstructing step, and, in the case where the gesture event is selectedin said selecting step, erasing the one or more mouse events included insaid held set at the time when the gesture event is selected in saidselecting step.
 13. The operation accepting method according to claim 8,wherein said browsing step further includes a determining step of, inthe case where said gesture event is selected, determining a displayarea, in said display image, to be displayed on said display portion byperforming one of said plurality of types of processes that correspondsto said selected gesture event.
 14. The operation accepting methodaccording to claim 8, further comprising an associating step of, at thestage when said browsing program is installed, associating each of saidplurality of types of processes determined by said browsing program withone of a plurality of gesture events in correspondence with saidbrowsing program, wherein said gesture event determining step includes astep of determining said gesture event on the condition that the gestureevent corresponding to the set of one or more mouse events generated insaid mouse event generating step is associated with one of saidplurality of types of processes in said associating step incorrespondence with said browsing program.
 15. A non-transitorycomputer-readable recording medium encoded with a browsing programperformed by a computer which controls a data processing apparatus, saiddata processing apparatus including a display portion capable ofdisplaying an image, a position detecting portion capable of detectingone or more positions designated by a user on a display surface of saiddisplay portion, a mouse event generating portion operable, in responseto detection of one or more positions by said position detecting potion,to generate one or more mouse events each including a piece ofpositional information indicating the corresponding one of said one ormore positions, and a gesture event determining portion to determine oneof the plurality of gesture events on the basis of a set of one or moremouse events generated by said mouse event generating portion, theprogram causing said computer to perform: a browsing step of executing abrowsing program, externally acquiring content described in a markuplanguage, and displaying on said display portion at least a part of adisplay image corresponding to said acquired content, and further, inresponse to input of one of a plurality of gesture events correspondingrespectively to a plurality of types of processes determined by saidbrowsing program, performing the one of said plurality of types ofprocesses that corresponds to said input gesture event, said browsingstep including a display image generating step of, in the case wheresaid content includes a description for displaying an embedded image tobe generated as an application program different from said browsingprogram is executed, generating a display image in which the embeddedimage generated as said application program has been executed isarranged in an embedded area determined by said content, and a selectingstep of, in the case where the position indicated by one of the one ormore mouse events on the basis of which the gesture event has beendetermined by said gesture event determining portion falls within saidembedded area, selecting either the gesture event determined by saidgesture event determining portion or the set of the one or more mouseevents on the basis of which the gesture event has been determined, onthe basis of the type of said determined gesture event.
 16. Thenon-transitory computer-readable recording medium encoded with thebrowsing program according to claim 15, wherein said selecting stepincludes a mouse event selecting step of, in the case where the gestureevent determined by said gesture event determining portion is not of aprescribed type, selecting the one or more mouse events on the basis ofwhich the gesture event has been determined by said gesture eventdetermining portion, and a gesture event selecting step of, in the casewhere the gesture event determined by said gesture event determiningportion is of said prescribed type, selecting the gesture eventdetermined by said gesture event determining portion.
 17. Thenon-transitory computer-readable recording medium encoded with thebrowsing program according to claim 15, wherein said gesture eventdetermining portion decides not to determine a gesture event in the casewhere a spacing between a first position indicated by a first mouseevent generated by said mouse event generating portion and a secondposition indicated by a second mouse event generated within a prescribedtime from the generation of said first mouse event is less than aprescribed distance.
 18. The non-transitory computer-readable recordingmedium encoded with the browsing program according to claim 17, whereinsaid browsing step further includes a holding step of holding a set ofone or more mouse events generated by said mouse event generatingportion until a gesture event is determined on the basis of the set bysaid gesture event determining portion or until it is decided not todetermine said gesture event by said gesture event determining portion,and an instructing step of, in the case where said set of one or moremouse events is selected in said selecting step, outputting the one ormore mouse events included in said held set to the task of executingsaid application program to cause the task to update the embedded imagedisplayed in said embedded area.
 19. The non-transitorycomputer-readable recording medium encoded with the browsing programaccording to claim 18, wherein said browsing step further includes anerasing step of, in the case where said set of one or more mouse eventsis selected in said selecting step, erasing the one or more mouse eventsincluded in said held set after the one or more mouse events included insaid held set are output to the task of executing said applicationprogram in said instructing step, and, in the case where the gestureevent is selected in said selecting step, erasing the one or more mouseevents included in said held set at the time when the gesture event isselected in said selecting step.
 20. The non-transitorycomputer-readable recording medium encoded with the browsing programaccording to claim 15, wherein said browsing step further includes adetermining step of, in the case where said gesture event is selected,performing one of said plurality of types of processes that correspondsto said selected gesture event to thereby determine a display area, insaid display image, to be displayed on said display portion.
 21. Thenon-transitory computer-readable recording medium encoded with thebrowsing program according to claim 15, causing said computer to furtherperform an associating step of, at the stage when said browsing programis installed, associating each of said plurality of types of processesdetermined by said browsing program with one of a plurality of gestureevents in correspondence with said browsing program, wherein saidgesture event determining portion determines said gesture event on thecondition that the gesture event corresponding to the set of one or moremouse events generated by said mouse event generating portion isassociated with one of said plurality of types of processes in saidassociating step in correspondence with said browsing program.